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METHOD FOR DRIVER ASSISTANCE AND DRIVER ASSISTANCE DEVICE ON THE 

BASIS OF LANE INFORMATION 

Background Inf ormat ion Field of the Invention 

The present invention relates to a method for driver assistance 
and a driver assistance device which operates on the basis of lane 
information . 

5 Background Inf ormat ion 

Driver assistance systems which operate on the basis of lane 
information are known in the related art. An example of such a 
driver assistance system is a warning system which warns the driver 
upon departing from the lane and/or upon imminent departure from 

■ 

10 . the lane. For example, published European patent document EP 
1074430 Al discloses a system of this type, in which the road 
surface (lane) on which the vehicle moves is established using 
image sensor systems^ and the driver is warned when the vehicle 
departs from this lane and/or threatens to depart from this lane. 

15 Furthermore, additional driver assistance systems of this type are 
known f rom di sclosed in published German Patent Appl ioat iona patent 
document 103 11 518.8, having the priority date of 04/30/02, and 
published German patent document 102 38 215.8, having the priority 
date of 06/11/02 (not prior publicat iono) . In these caaca systems , 

2 0 image sensor systems which are installed in the vehicle and which 
record the scene in front of the vehicle are used to detect the 
lane. The boundaries of the lane^_ and therefore the lane itself^ 
are ascertained from the recorded images of the lane boundary 
markings. Ascertaining the lane is accordingly essentially a 

25 function of the existing visibility, the known systems having to 
be shut down early in the event of poor visibility. 
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An example of the t recognition and modeling of lane boundary 
markings from video images, lane width, lane curvature, curvature 
change, and lateral offset of the vehicle, among other things, 
being ascertained as the model parameters, is known fro m described 
5 in German patent document DE 19627 938 A3b. 

Advantages of — fehe — Invontio n Summary 

By using further information in addition or alternatively to the 
lane boundary markings, from which the variables describing the 
course of the road (lane) are derived, the availability of a driver 

10 assistance system based on lane information is significantly 
increased in accordance with the present invention . It is 
particularly advantageous that the driver assistance system is 
also available if the lane boundary markings are no longer reliably 
recognizable. This is significant above all in poor weather 

15 conditions, for example, a wet road surface, a snow-covered road 
surface, etc. , or in the event of poorly visible and/or nonexistent 
lane boundary markings. 

It is particularly advantageous that in addition to the lane 
boundary markings or even instead of these, other information may 

2 0 be used individually or in any arbitrary combination in each case 

for lane identification, such as the trajectory of one or more 
preceding vehicles, the tracks of one or more preceding vehicles 
in the event of rain or snow, for example, the trajectory of one 
or more oncoming vehicles, and the course of road boundaries such 
25 as guard rails, curbs, etc. Lane information may also be derived 
(estimated) from this data, which forms the lane information (lane 
data) for the driver assistance system instead of or together with 
the lane information ascertained from the lane boundary markings. 
Lane identification thus becomes more reliable, in particular if 

3 0 the actual lane boundary markings are no longer sufficiently 

recognizable . 

It is particularly advantageous that this is performed solely on 
the basis of the signals of the image sensor system, without 
additional hardware . 
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It i& particularly advantageous that quality indices for the lane 
data detection are determined from the image contrast, for example, 
using which the particular lane data which has been ascertained 
may be weighted and taken into consideration during the merger of 
5 the lane data provided to the driver assistance system from the 
individual lane data. It is particularly advantageous in this 
context that forming an overall quality index for the lane data 
detection from the individual quality indices is provided, the 
driver assistance system being shut down if this overall quality 
10 index falls below a specific value. It is also advantageous if the 
quality index is derived from a comparison of the estimate with 
the measurement, the deviation of the measured points from the 
estimated line (variance) being used in particular , for example . 

Furthermore, it is advantageous that by increasing the 
15 availability of the driver assistance system even in poor weather 
conditions in particular , the driver support assistance system 
functions precisely when the driver particularly needs the 
oupport assistance . The driver is significantly relieved by the 
operation of the driver assistance system during poor weather 
20 conditions in particular. 

When ascertaining the lane data from information other than the 
lane boundary markings (which is also referred to in the following 
as lane data estimate) , data of a global positioning system and/or 
data of the navigation map and/or immobile objects standing next 
25 to the road, which are classified by the video sensor, are 

particularly advantageously analyzed for the plausibility check 
of the lane data. Lane data acquisition (lane data estimate) thus 
becomes more reliable. 

It is also particularly advantageous that in the event of loss of 
30 data values, for example, the values for the lane width, values 
before the loss or empirical values and/or average values are used 
for these variables in the lane data estimate. Therefore, the 
function of the lane data estimate is also ensured under these 
circumstances . 

3 5 Further advantagca rcoult — from the — following dcocription of 
exemplary embodiment a — and — from the — dependent — claimo . 
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Drawing 

The proocnt — invention ia deocribed in greater detail — in the 
following on the baoio of the embodiments iliuotratcd in the 
drawing . Brief Description of the Drawings 

5 Figure 1 shows a block diagram of a driver assistance system-? — in 
particular for driver warning and/or for response if the vehicle 
threatens to depart from the lane. 

Figure 2 shows a schematic flow chart o# il lust rating a first 
exemplary embodiment for providing the lane data information. 

10 Figures 3 through 5 show various flow charts which 

represent illustrating operation of a second example embodiment 
e# f or the measurement and estimate of lane data and its analysis 
in the driver assistance system. 

Description of — the Exemplary E m b odimcnt oDetailed Description 

15 Figure 1 shows a device which is used for warning the driver and/or 
for response if the vehicle departs from the lane. A control unit 
and/or analyzer unit 10, which has an input circuit 12, a 
microcomputer 14, and an output circuit 16, is shown. These 
elements are connected to one another using a bus system for mutual 

20 data exchange. Input lines from different measuring devices are 
connected to input circuit 12, via which the measured signals 
and/or measured information are transmitted. A first input line 
20 connects input circuit 12 to an image sensor system 22, which 

* 

is situated in the vehicle and which records the scene in front 
25 of the vehicle. Corresponding image data is transmitted via input 
line 20. Furthermore, input lines 24 through 26 are provided, which 
connect input circuit 12 to measuring devices 3 0 through 34. These 
measuring devices are, for example, measuring devices for 
measuring the vehicle velocity, for detecting the steering angle, 
30 and for detecting further operating variables of the vehicle which 
are significant in connection with the function of the driver 
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assistance system. Furthermore, map data and/or position data of 
the vehicle is supplied via these input lines. At least one warning 
device 38 is activated via output circuit 16 and output line 36, 
such as a warning light and/or a loudspeaker for an acoustic warning 
5 and/or for a voice output and/or a display for displaying an image, 
with the aid of which the driver is informed and/or warned of the 
imminent lane departure. A haptic warning (e.g., steering wheel 
vibration) may also be provided. In another exemplary embodiment, 
a servo system 42 is alternatively or additionally activated via 
10 output circuit 16 and an output line 40, which automatically guides 
the vehicle back into the lane by intervening in the steering of 
the vehicle and thus prevent a preventing it from departing the 
lane . 

Lane d ata i s ascertained — in an embodiment — corresponding to the 

15 relate d art cited at th e bcginning ln ascertaining the lane data 
convent ional ly , lane modeling parameters bcing are ascertained by 
analyzing the detected image according to an imaging specification 
which includes the camera data and being adapted to the measured 
image. Thus, the driver assistance system analyzes the image 

2 0 detected by the image sensor and ascertains objects in the image, 
in particular the lane boundary markings (e.g., center lines, etc.). 
The courses of the ascertained lane boundary markings (left and 
right) are then mathematically approximated by functions, e.g., 
as the clothoid model, approximated by a second-order polynomial, 

25 for example. Parameters of these equations are, for example, 
curvature and curvature change, and the distance of the host 
vehicle to the boundary markings on the right and on the left . 
Furthermore, the angles between the tangents of the calculated lane 
and the direction of movement of the host vehicle may be ascertained. 

30 The lane information ascertained in this way is then supplied to 
the driver assistance system, which recognizes an imminent lane 
departure and warns the driver and/or initiates countermeasures 
at the suitable instant on the basis of the actual trajectory 
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(trajectories) of the vehicle (determined on the basis of the 
. ■' » 

steering angle, for example) . 

As long as the lane boundary markings are clearly recognizable in 
the recorded image, the calculation of thio the lane data as 
5 described above is precise and reliable. In the event of poor 
weather conditions and/or poor visibility and/or poorly visible 
or nonexistent lane boundary markings, the method described above 
may be imprecise and/or may not be able to provide a result . Systems 
operating on the basis of the lane data would then have to be shut 

10 down in such situations. Therefore, in accordance with the present 
invention, an extension of the lane data detection and thus an 
extension of the driver assistance system connected thereto is 
described in the following, which allows further operation of the 
driver assistance system even in the event of poor weather 

15 conditions and/or poorly visible or nonexistent lane boundary 

markings by calculating a lane (estimating a lane) on the basis 
of information from the recorded image other than the lane boundary 
markings, while incurring no additional outlay in hardware 
equipment being incurrod costs . 

20 A flow schematic chart is illustrated in Figure 2, which represents 
a first exemplary embodiment in regard to the above-mentioned 
extension of the lane data detection. The flo w schematic chart 
represents the program running on the microcomputer in control 
and/or analyzer unit 10 in this case. 

25 The starting point is an image sensor 2 00 which is installed in 
or on the vehicle and records the scene in front of the vehicle. 
Appropriate image signals are relayed via lines 202 to analyzer 
unit 10. In addition to the lane data calculation on the basis of 
lane boundary markings described above, analyzer unit 10 analyzes 

30 the transmitted images as follows. 

First, as described above, the lane boundary markings in the image 
are recognized in module 204 and then the lane data is calculated 
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in module 206. In the illustrated exemplary embodiment, the courses 
of the tracks of one or more preceding vehicles, which are visible 
on a wet road surface, in snow, etc. , for example, are ascertained 
in a second module 208 . This is achieved through analysis and object 
5 recognition in the image on the basis of the grayscale values, 
for example (e.g. , gradient analysis) . Within this representation, 
objects are also understood as the lane boundary marking and/or 
the road boundary construction (guard rails, etc.) . The track 
recognized in this way is then described mathematically using the 
10 cited parameters as described above. The lane width (estimated, 
from map data, etc.) is also considered in this case. 

The trajectory of one or more preceding vehicles and/or oncoming 
vehicles is recorded in module 210 on the basis of sequential images. 
This is performed through object recognition and object tracking 

15 in the individual images, the parameters being derived from the 
changes in the object. The lane width and/or the offset between 
oncoming traffic and traffic on the current lane are considered 
as estimated values. Stationary objects on the road boundary, such 
as guard rails, are analyzed and the trajectory is determined on 

20 the basis of this information in module 210 as an alternative or 
as a supplement . 

Furthermore, a quality index (e.g., a number between 0 and 1) for 
the particular lane data is ascertained on the basis of the images 
provided by the image sensor on the basis of the image contrasts 

25 in the area of the particular analyzed object, for example, and 
is also provided with all ascertained lane data. An alternative 
or supplementary measure for ascertaining the quality index is a 
comparison of the estimate with the measurement, the deviation of 
the measured points from the estimated line (variance) being used 

30 in particular. If the variance is large, a small quality index is 
assumed; if the variance is small, a high quality index is 
specified . 
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The additional lane data ascertained in this way is analyzed to 
form a set of estimated lane data, possibly considering the quality 
indices, in lane data estimate module 212 . In a prcf crrcd an example 
embodiment, this is performed by weighting the lane data 
5 ascertained in different ways using the assigned ascertained 

quality index and calculating the resulting lane data from this 
weighted lane data of the different sources, e.g., by calculating 
the mean value. A resulting quality index is thus determined. 

In a prcfcrrod an example embodiment, a global positioning system 
10 and/or map data 214 is also provided, whose information is 

evaluated within the lane data estimate as a plausibility check. 
For example, it is checked on the basis of this map data and/or 
positioning data whether or not the ascertained lane data 
corresponds to the map data within the required precision. In the 
15 latter case, a quality index for the lane data is determined as 
a function of a comparison of the estimated data with the map data, 
the quality index being smaller at larger deviations than at 
smaller deviations. If specific lane data cannot be ascertained 
from the available data, experiential ( standard) values or values 

2 0 before the loss of the information are used. For example, if the 

width of the lane cannot be ascertained from the currently 
available information, either experiential values for the lane 
width or the values established for the lane width during the last 
lane data estimate are used. 

25 The lane data estimated in this way is then supplied to a lane data 
merger 216, in which the estimated lane data having the resulting 
quality index and the calculated lane data (also having a quality 
index) on the basis of the lane boundary markings are combined into 
the lane data used for the function. The data merger is also 

3 0 performed here while taking the quality indices into consideration, 

for example, by discarding the corresponding data in the event of 
a very low quality index, or in the event of a very high quality 
index of one of the calculation pathways, using only this data and 
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calculating a mean value in the intermediate area. A resulting 
« ■ 

quality index may also be ascertained accordingly. 

The lane data ascertained in this way is provided to the following 
analyzer unit, which then warns the driver upon imminent lane 
5 departure on the basis of this lane data, for example. 

A further exemplary embodiment of the ouggcotcd approach the driver 
assistance system and method is illustrated a-e in connection with 
flow charts in Figures 3 through 5. The flow charts represent 
programs or parts of programs for the microcomputer which is 
10 situated in analyzer unit 10. 

Figure 3 shows an example which represents an analysis of the 
ascertained lane data using the example of a system for warning 
before departing the lane. First, in step 300, the lane data which 
is measured and/or estimated or derived from a merger of the two, 

15 and/or the shutdown information (see below) is input. In step 301, 
it is checked whether there is shutdown information. If so, the 
program is ended and executed again at the next instant. Otherwise, 
in step 302, the actual trajectories of the vehicle and, therefrom, 
the future course of the vehicle (left and/or right vehicle side) 

20 are calculated as a mathematical function (with the assumption that 
the vehicle will maintain the current course, possibly taking 
typical driver reactions into consideration) on the basis of 
vehicle variables such as steering angle, yaw rate, lateral 
acceleration, vehicle geometry data, etc. Then, in step 304, 

25 imminent lane departure is derived (intersections of the 

mathematical functions in the future) by comparing the ascertained 
lane data and the future course of the vehicle (on one or both lane 
sides) . If this is the case, according to step 306, the driver is 
acoustically and/or optically and/or haptically warned, and, in 

3 0 one exemplary embodiment, the vehicle is possibly kept in the lane 
through steering intervention. If the comparison shows that lane 
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departure* is not to be feared, the warning and/or the action 
described does not occur. 

Figure 4 ohowo illustrates a method for ascertaining lane data on 
the basis of available estimated lane data. First, in step 400, 
5 the lane boundary markings are recognized from the image detected 
by video sensor 200 using methods of image analysis, e.g., on the 
basis of the image contrasts and comparison with stored models. 
Furthermore, in step 4 02, a quality index is calculated from the 
contrast of the image, in particular from the contrast in the area 

10 of the lane boundary markings, and/or the variance of the measured 
values and the estimated values. In the prefcrrcdan exemplary 
embodiment, this quality index is a value between 0 and 1, the 
quality index being higher the higher the contrast of the image 
and/or the smaller the variance. In following step 404, the lane 

15 data is then calculated on the basis of the recognized lane data 
markings, in particular, a second-order polynomial is produced and 
the lane parameters of curvature and curvature change and distance 
on the left and right to the host vehicle are calculated, so that 
lane data for the left and right boundaries is provided. In step 

20 406, the lane data from the lane data estimate (which is also 
provided for right and left) and the quality index connected 
thereto are input. In step 408, the merger of this lane data is 
then performed for each side individually to produce the resulting 
lane data. This is performed while taking the established quality 

25 indices into consideration. Thus, for example, with a high quality 
index (for example, > 0.75) in the detection of the lane boundary 
markings, the lane data estimate is not used at all. There may also 
be provided other exemplary embodiments in which, vice versa, with 
a high quality index of the lane data estimate and a low quality 

30 index of the lane boundary markings recognition (< 0.3, for 

example) , the lane data from the estimate is used. In other cases, 
the merger is performed by calculating a weighted mean value from 
the lane data available, for example, the weighting being performed 
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on the basis of the quality indices. A final resulting quality index 
is ascertained from the quality indices, as with the lane data. 
In step 410, it is checked whether this resulting quality index 
has reached a specific value, such as 0 . 5 . If not, instead of the 
5 lane data, shutdown information is sent to the following systems 
in step 412, in such a way that reliable lane data cannot be 
ascertained. Otherwise, the resulting lane data is relayed to the 
following subsequent application (step 414) . 

Figure 5 shows a flow chart which outlines an exemplary 

10 cmbodimcnt method for ascertaining the estimated lane data. The 
image ascertained by the video sensor is also analyzed in first 
step 500 here. Different objects in the image are recognized, such 
as preceding vehicles, oncoming vehicles, or stationary objects 
such as guard rails which identify the road boundary, and 

15 immob i le stationary objects outside the road, such as trees, etc. 
The analysis of the image and the object recognition and 
classification of the objects ieare performed in accordance with 
an appropriate image analysis method, e.g., on the basis of the 
contrasts existing in the image and contour comparisons. In 

20 following step 502, quality indices for the object recognition are 
ascertained from the contrasts of the image details in which the 
ascertained objects lie, and/or from the variance of the 
corresponding measured and estimated values. Every recognized 
object is provided with a corresponding quality index (e.g., a 

2 5 value between 0 and 1) in this case. 

In subsequent step 504, lane data is derived from the objects. For 
preceding vehicles or oncoming vehicles, this is performed by 
analyzing sequential images, from which the movement of the 
vehicles, their direction, and their trajectories in the past may 
30 be ascertained. The trajectories ascertained in this way are then 
used for determining a lane course. The oncoming traffic is 
suitable for this purpose in particular, whose trajectory in the 
past represents the lane to be traveled by the vehicle. Taking the 
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lateral distance between the preceding vehicles and oncoming 
vehicles into consideration, the course of the current lane is 
ascertained. The above-mentioned lane data is then established in 
turn from the trajectory and an assumed or ascertained lane width. 

5 In rain or poor visibility or even in snow, the track of the 

preceding vehicle which is then visible may be analyzed from the 
recorded image. Trajectories may be calculated from the image 
analysis which approximately correspond to the course of the lane . 
boundary markings when an assumed lane width is taken into 
10 consideration. The lane data is also represented here as a 

mathematical function from the objects recognized in the image. 

As a further possibility, immobilc stationary objects may be 
analyzed to estimate lane data, in particular guard rails or other 
delimitations which delimit the road on at least one side. The 
15 course of these delimitations may be analyzed in the image and a 
trajectory may be calculated therefrom. Taking typical lateral 
distances and lane widths into consideration, lane data (right and 
left) may then be ascertained. 

As noted above, a quality index is assigned to every ascertained 
20 object, which is correspondingly included with the road data 
ascertained on the basis of this object. Furthermore, 
immobilc stationary objects, which are classified by the video 
sensor and mark areas which may not be traveled, are used for the 
plausibility check of the estimated lane. If it is recognized that 
25 the estimated lane is located in the area of such 

immobilc stationary objects, an erroneous lane estimate is to be 
assumed . 

The ascertained lane data and the resulting quality index are then 
forwarded for further analysis ( comparc see Figure 4) . 

30 In a prcfcrrcd an example embodiment, the lane estimate is only 
performed when poor weather conditions have been recognized, while 
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in good weather conditions and good visibility, the estimate is 
dispensed with. Poor weather conditions are recognized in this case 
if the windshield wipers are active beyond a specific mcaouro rate 
and/or if a rain sensor recognizes rain and/or if the video sensor 
ascertains a low visibility range. 

Furthermore, in one exemplary embodiment, the quality of the lane 
estimate is reduced if it is recognized that the preceding vehicle 
is turning off or changing lanes. 
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Abatract ABSTRACT 

A method for driver assistance and a driver assistance device which 
operates on the basis of lane information are dcocribcd . 
^ghe provided, which lane information is ascertained from an image 
5 recorded by an image sensor and/or estimated on the basis of objects 
in this image depending on the weather conditions. 

(Figure — 34- 
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